Papers with unigram LM method
Byte Pair Encoding is Suboptimal for Language Model Pretraining (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization is a popular language model that can be used to segment text. |
| Approach: | They analyze differences between byte-pair encoding (BPE) and unigram LM tokenization methods to find subword units that align more closely with morphology. |
| Outcome: | The proposed method recovers subword units that align more closely with morphology and avoids problems stemming from BPE’s greedy construction procedure. |